Anyone having EPB issues after their big "network upgrade" recently
From: Lynn Dixon ------------------------------------------------------ I've been a LONG time EPB residential customer and have never had any issues.....until a few weeks ago. I received an email saying EPB was planning "major network upgrade" and that there would be some downtime associated. No biggie, seems reasonable to me. Then they did the "upgrade", I had a few hours of downtime and then back online. After playing around I noticed a LOT of issues that I have NEVER had before. First off, my public IPv4 address had changed. I've had the same address for the 8 years I've been an EPB customer, but oh well. Thats the smallest of the issues. Then I discovered that my daughter's Nintendo Switch is having a LOT of problems with matchmaking, so upon checking, we are now getting a NAT rating of "D" on Nintendo, which is not good. Same goes with Xbox Live and Playstation networks. Finding a match is very very slow and almost never finds matches. I also noticed a lot of the services I use when I'm on the road connecting back home simply don't work, even after I updated the IP's and Dynamic DNS settings to reflect my new public IP. I can't access my OpenVPN server at home any more. At all. This has worked without fail for many years. Now I can't access it on my home pfSense router. Nothing has changed on my router. I even rebooted it a few times after the EPB "upgrade" and nothing. I also can't access my WireGuard server at home. Again, nothing has changed in the server itself. And I have made sure to update and verify all the clients and peers are set to the proper public IPv4. I am also having a lot of high ping times to services even close (to my servers co-located in Peace Communications Datacenter). Ping times in excess of 100ms when it was usually around 8ms before their "upgrade". Not only that, but DNS lookups to 1.1.1.1 and even 8.8.8.8 are just really, REALLY slow. I have also noticed that my fastest speed is topping out around mbps, when it would routinely be in the 900's consistently. Has anyone else noticed the same after their huge core network upgrade that happened a month or so ago?=============================================================== From: John D ------------------------------------------------------ I haven't worked at EPB in years (I don't even live in the South anymore!) so I can't really comment on much but a lot of this sounds like you might be behind CGNAT? Did you check the IP address in your PFSense and see if it's showing a 100.64.0.0/10 IP? EPB should be able to disable it for free if you give them a ring or talk to them on chat. It really shouldn't affect latency or anything of that sort though - but the issue with online gaming and not being able to access your OpenVPN server remotely def sound like carrier grade NAT. On Mon, Mar 27, 2023 at 7:35=E2=80=AFPM Lynn Dixon wro= te: s d s ot n
=============================================================== From: Phil Sieg ------------------------------------------------------ Lynn, Not having these problems but I have a couple of thoughts: 1=2E Yo= u may be on Carrier grade NAT=2E We see this from time to time with our cus= tomers (other carriers not epb)=2E Call them and escalate to tech support a= nd find out=2E=C2=A0 2=2E My second thought no longer makes sense: that t= hey need to clear your MAC cash (I used to run into this with them back whe= n I changed routers a lot) but that hasn't changed in your use case=2E=C2= =A0 You likely need to speak to a fairly high level tech there and hash i= t out=2E=C2=A0 My guess is they have instituted some "bandwidth efficiencie= s tomfoolery" that wouldn't affect your average home user the way it affect= s a geeky IT professional home user=2E =E2=81=A3Phil Sieg Founder ring-u = llc www=2Ering-u=2Ecom phil@ring-u=2Ecom Phone: 423=2E567=2E4888 Mobile: = 423=2E331=2E0725 "The computer is the most=C2=A0remarkable tool that we've= ever=C2=A0come up with=2E It's the equivalent=C2=A0of a bicycle for our mi= nds=2E" Steve Jobs, 1955-2011=E2=80=8B On Mar 27, 2023, 10:35 PM, at 10:3= 5 PM, Lynn Dixon wrote: esidential customer and have never had any w weeks ago=2E I received an email saying EPB was upgrade" and that there would be some downtime seems reasonable to me=2E of downtime and then T of issues that I have dress had changed=2E I've had an EPB customer, but oh well=2E hen I discovered that my daughter's Nintendo Switch is having a LOT of blems with matchmaking, so upon checking, we are now getting a NAT of "D" on Nintendo, which is not good=2E Same goes with Xbox Live ystation networks=2E Finding a match is very very slow and almost finds matches=2E I also noticed a lot of the services I use when I'm on e P's ess my OpenVPN server at home any more=2E At all=2E This has out fail for many years=2E Now I can't access it on my home pfSense r=2E Nothing has changed on my router=2E I even rebooted it a few = server at home=2E Again, nothing has d I have made sure to update and to the proper public IPv4=2E to services even close (to my atacenter)=2E Ping times in s before their "upgrade"=2E =2E1 and even 8=2E8=2E8=2E8 are just really, noticed that my fastest speed is topping out around mbps, routinely be in the 900's consistently=2E me after their huge core network upgrade
=============================================================== From: Dave Brockman ------------------------------------------------------ It sounds an awful lot like you are now behind CGNAT. Post your WAN IP=20 (or unicast if you prefer). -dtb was=20 k=20 e=20 =A0 I've=20 OT of=20 lmost=20 m=20 d=20 . as=20 ome=20 booted it a=20 as=20 verify=20 =20 s in=20 =20
=============================================================== From: Lynn Dixon ------------------------------------------------------ It was CGNAT! Called support and when I briefed her on what I was experiencing, she immediately said "You're on CGNAT, I can move you off". Apparently during the core network upgrade, the sticky DHCP reservations cache was flushed and somehow my pfSense box's MAC was put into the CGNAT pool. She moved me out and a quick reboot of the router and all was working like normal again. Its refreshing that even the first level support folks at EPB can get technical and fix power-user issues quickly. On Mon, Mar 27, 2023 at 11:05=E2=80=AFPM Dave Brockman = wrote: =3D20 =3D20 EPB =3D =3D20 =3D hav=3D =3D20 a L=3D =3D20 =3D20 nd a=3D =3D =3D 2=3DA0IP=3D is h=3D my h=3D even re=3D ng h=3D and =3D =3D time=3D =3D 0 =3D20 =3D20
=============================================================== From: Sudo Bash ------------------------------------------------------ They've been using CGNAT for a few years now...
=============================================================== From: Michael Harrison ------------------------------------------------------ My bet is you got "CGN'd" as in Carrier Grade Nat. An easy way to tell used to be to compare your local external/public address on your router with what the internet sees. In some cases, it varies depending on the protocol. YMMV. On Mon, Mar 27, 2023 at 10:35=E2=80=AFPM Lynn Dixon wr= ote: s d s ed s ot n at
=============================================================== From: Joe Freeman ------------------------------------------------------ Lynn- Have you reached out to tech support @ 423-648-1372? If you hit me off the list, I can look into what PON you're on and when you got moved. I'd also like to know when these issues started as we've been doing a lot of upgrades over the last couple of years, and none of the issues you're describing have reached me or my team. This is not the experience we want for our customers, so I definitely want to get one of my engineers looking into this. Thanks- Joe On Mon, Mar 27, 2023 at 10:45=E2=80=AFPM John D = wrote: ) e d rote: R e ts nd r e 's Not
=============================================================== From: Dave Brockman ------------------------------------------------------ =20 =20 s put=20 outer=20 Something changed with one of their upgrades, they don't do DHCP=20 reservations any more, so any MACs assigned to it went to the default=20 pool, which is now CGNAT. I'm not sure exactly what they do now for us,=20 because my WAN IP definitely changes, but I always get a public IP=20 address when it does. They were well prepared for this when they rolled it out. For anyone=20 else who might experience the same, if your assigned WAN IP is between=20 100.64.0.0 to 100.127.255.255, all you have to do is call EPB and tell=20 them VPN to work doesn't work, you think have CGNAT addressing, can you=20 please move me off? -dtb
=============================================================== From: Jason Brown ------------------------------------------------------ Do give Tech Support a shout, EPB seems to actually care about their quality. Thanks Joe! --Jason On Tue, Mar 28, 2023 at 9:10=E2=80=AFAM Joe Freeman wrot= e: e g e and wrote: k ER he ats and er he P's a y Not --=20 Jason Brown
=============================================================== From: Lynn Dixon ------------------------------------------------------ Joe, I chatted with tech support last night and within a few minutes she had me out of the CGNAT pool and back to normal. They did a great job and that was also her first suggestion of "Looks like your in the CGNAT". These issues started around the second week of March-ish. I can't pinpoint exactly because I don't always use VPN/Wireshark and sometimes we didn't do a lot of online gaming those first couple weeks of march. I'll send a unicast email to you with my current IPv4 address after the move out of the NAT. On Tue, Mar 28, 2023 at 9:10=E2=80=AFAM Joe Freeman wrot= e: e g e and wrote: k ER he ats and er he P's a y Not
=============================================================== From: Joe Freeman ------------------------------------------------------ It is entirely possible this is a CGNAT issue and all we need to do is flip a bit, reset the ont, and have you reset your router or do a DHCP release/renew. As part of this upgrade a significant number of customers were moved into CGNAT pools to try to save public ipv4 address space. I'm not sure if anyone pays attention to the cost of IPv4 space out there in the market today, but IPs have gotten really expensive. However, if CGNAT is causing problems, we can flip a bit to move you back to a public IP, which will still be dynamically assigned out of a pool of addresses so there's no guarantee that you'll keep it forever. If there is something else going on, I really need to know about it as we're more than halfway through this phase of upgrades, and have already started the next phase (replacing the ONT's on the side of the house/building). So if anyone is having issues, please let me know either at this address, or by opening a support case with tech support - 423-648-1372 for residential, 423-648-1500 for businesses. Thanks- Joe On Tue, Mar 28, 2023 at 8:25=E2=80=AFAM Michael Harrison wrote: wrote: R e ts nd r e 's es en
=============================================================== From: Unkmar ------------------------------------------------------ This has not happened to me so far. I will now be on the look out for it. When EPB was initially setup here, I had to call to get the double NATti g bypassed. (CGNAT?) It was quick, easy, and no issues. I fully understand why they do it. Effectively, they must be providing two IPv4 addresses here. One for the TV service and another for me to have an outward facing internet IP. I see the issue as ironic, since I don't get an IPv6 address from my ISP. So, I don't have one. It feels like American ISPs just keep digging the IPv4 hole deeper. PS: Just my two cents worth. -- Lucius L. Hilley III wrote: k ER he er he y
=============================================================== From: Dave Brockman ------------------------------------------------------ I have never encountered an issue with having to be removed from=20 CGNAT more than once. If you did it at install, you should be good. I=20 haven't actually looked, but I doubt there is a need for a public IP on=20 the TV side, maybe Joe will share more details. EPB has had IPv6 on their network for some time, but I still do not=20 receive an IPv6 address on either my business or residential circuits.=20 I had it all set up on my last datacenter build with an HE tunnel, but=20 on last refresh, I just tore it all down. It was fun to play with, but=20 didn't really provide any real value to maintain the dual-stack (twice=20 the device config, twice the firewall rules, etc). I thought that EPB=20 turned up IPv6 for one of their customers using them for transit/DIA,=20 but I have slept and drank lots of good whiskey since I had that=20 conversation. I *think* I could turn it up on my BGP session with them,=20 but until I have a real use case to get IPv6 space from ARIN, I'll just=20 continue to ride the IPv4 wave. (and yes, I know I'm spoiled, I have my=20 own assignment). -dtb 4=20 t=20 a y u f ever. e
=============================================================== From: Joe Freeman ------------------------------------------------------ IPv6 availability depends on which type of OLT node you're on. The big moves that are happening right now are moving every one off the legacy nodes that are not IPv6 capable. Once you've been moved over to the newer hardware, IPv6 should be available via DHCPv6 and prefix delegation. If you've already been moved over and aren't getting IPv6 using DHCPv6 and Prefix Delegation, please contact tech support (423-648-1372). Thanks- Joe g m k f , ck f fy y ,
=============================================================== From: David White ------------------------------------------------------ IPv6 is working perfectly fine for me. As Joe mentioned, you have to setup DHCPv6 Prefix Delegation. I'd be happy to share my Ubiquiti config for this .... On Tue, Mar 28, 2023 at 3:40=E2=80=AFPM Joe Freeman wrot= e: d . g g s s 'm n pool y for e of e e my n y, e --=20 David White
=============================================================== From: Dave Brockman ------------------------------------------------------ Hi Joe, How would one know which type of OLT node we are on? I am the technical contact for so many customers/circuits, and your upgrades are (mostly) hitless (as far as requiring any support from me after the fact) that I ignore the upgrade notifications, and have for years. No one dies if my circuit or one of my client's circuits drop for an hour at 2AM on a weekend. Kudos for that. Just to clarify, do you also use RA for gateway assignment in addition to DHCPv6? What is the size of your delegated PD subnet assignment? Are PD prefix hints honored (to any degree)? With Gratitude, Dave Brockman Senior Network Engineer Gig City Cloud, LLC
=============================================================== From: Dave Brockman ------------------------------------------------------ If that's an ER config, please. :) Not seeing anything on the house circuit, but I just noticed the datacenter config has IPv6 disabled. Not really sure I'm ready to start down this rabbit hole again, but I'm going to pour another bourbon and think about it. :) With Gratitude, Dave Brockman Senior Network Engineer Gig City Cloud, LLC
=============================================================== From: Joe Freeman ------------------------------------------------------ Dave- Hit me up off the list. Joe On Tue, Mar 28, 2023 at 7:45=E2=80=AFPM Dave Brockman = wrote:
=============================================================== From: Kenneth Vandergriff ------------------------------------------------------ Check out tail scale. A VPN with a free version that solves the problem caused by many of the ISP's double natting. On Tue, Mar 28, 2023 at 9:10=E2=80=AFAM Joe Freeman wrot= e: e g e and wrote: k ER he ats and er he P's a y Not --=20 * kvandergriff@gmail.com *
=============================================================== From: Dave Brockman ------------------------------------------------------ It's wireguard, just use that if you have to VPN. The real solution (in=20 this case especially) is get yourself removed from the CGNAT pool. With Gratitude, Dave Brockman Senior Network Engineer Gig City Cloud, LLC roblem=20 t me n t h eived d ie, a LOT dress s , s t ttings n't ged on PB n, made s hat, ,